assessment of otoscopy: how does observation compare to a review of clinical evidence

background and purpose: to investigate how much the method of observation agrees with a standardised review of evidence of clinical examination, for the assessment of clinical otoscopic competence. methods: 65 medical students took part in an objective structured clinical examination (osce) station using patients with real pathology. examiners assessed otoscopic competency in tympanic membrane examination solely by distant observation. an external examiner later reviewed candidates’ documented findings on a schematic drawing of the tympanic membranes. observed agreement of the two methods and cohen’s kappa coefficient were calculated. results: mean otoscopy scores for examiner 1 and examiner 2 were 67.7% and 29.4% respectively. there was a significant difference using the mann-whitney u-test. osce observation declared 47.7% of candidates (31/65) to be clinically competent. drawing-based analysis however deemed only 4.6% (3/65) to have achieved this competency. this represented more than a ten-fold overestimation of clinical competency by osce assessment. observed agreement between assessment methods was 59.6%. cohen’s kappa coefficient was 0.1. conclusions: osce observational assessment of otoscopic clinical competency correlates poorly with review of evidence from clinical examination. if evidence review is acceptable as a better marker for competency, observation should not to be used alone in osce assessment. evidence review itself is vulnerable to candidate guesswork. osce could possibly explore candidate demonstration with explanation of findings, by use of digital otoscopy offering a shared view of the tympanic membranes, as an improved standard of clinical competency assessment.